Method and apparatus for active reduction of mechanically coupled vibration in microphone signals

ABSTRACT

An integrated circuit includes a processor that upsamples a vibration signal. The processor determines a correlation value. The correlation value may be based on a microphone signal, the upsampled vibration signal, or both. The processor determines filter coefficients. The filter coefficients may be determined based on the correlation value being above a threshold. The filter coefficient may be based on the upsampled vibration signal. The processor filters the vibration signal based on the filter coefficients to remove a noise portion and obtain a processed microphone signal. The processor outputs the processed microphone signal.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/710,902, filed on Dec. 11, 2019, which is incorporated by referencein its entirety.

TECHNICAL FIELD

This disclosure relates to noise reduction for electronic devices.

BACKGROUND

Microphones in electronic devices are prone to detect unwantedstructural vibration noise in addition to detecting desirable acousticvibrations. Typical solutions to reduce unwanted structural vibrationnoise include the use of mechanical isolation devices such as dampeners.Mechanical isolation devices can add to the weight of the electronicdevices. In addition, mechanical isolation devices can limit the designof the electronic devices due to size constraints. It would be desirableto have a method and apparatus to reduce unwanted structural vibrationnoise in microphone signals using sensor data.

SUMMARY

Disclosed herein are implementations of a method and apparatus foractive reduction of mechanically coupled vibration in electronicdevices. In an aspect, an image capture device may include a microphone,a vibration sensor, and a processor. The microphone may be configured toobtain a microphone signal. The microphone signal may include anacoustic signal portion and a noise portion. The vibration sensor may beconfigured to obtain a vibration signal. The processor may be configuredto receive the microphone signal, the vibration signal, or both. Theprocessor may be configured to upsample the vibration signal. Theprocessor may be configured to determine a correlation value. Thecorrelation value may be based on the microphone signal, the upsampledvibration signal, or both. The processor may be configured to determinefilter coefficients. The filter coefficients may be referred to as a setof filter coefficients. The filter coefficients may be determined basedon the correlation value being above a threshold. The filter coefficientmay be based on the upsampled vibration signal. The processor may beconfigured to filter the vibration signal based on the filtercoefficients to remove the noise portion of the microphone signal andobtain a processed microphone signal. The processor may be configured tooutput the processed microphone signal.

In another aspect, an image capture device may include a microphone, avibration sensor, and a processor. The microphone may be configured toobtain a microphone signal at a first sampling rate. The microphonesignal may include an acoustic signal portion and a noise portion. Thevibration sensor may be configured to obtain a vibration signal at asecond sampling rate. The second sampling rate may be less than thefirst sampling rate. The processor may be configured upsample thevibration signal. The processor may be configured to determine acorrelation value. The correlation value may be based on the microphonesignal, the upsampled vibration signal, or both. The processor may beconfigured to filter the vibration signal based on the filtercoefficients to remove the noise portion of the microphone signal andobtain a processed microphone signal. The processor may be configured tooutput the processed microphone signal.

In another aspect, a method may be implemented in an electronic deviceto reduce unwanted structural vibration noise in microphone signals. Themethod may include obtaining a microphone signal. The microphone signalmay include an acoustic signal portion. a noise portion, or both. Themethod may include obtaining a vibration signal. The method may includeupsampling the vibration signal. The method may include determining acorrelation value. The correlation value may be based on the microphonesignal, the upsampled vibration signal, or both. The method may includedetermining filter coefficients. The filter coefficients may be based onthe upsampled vibration signal. The method may include filtering thevibration signal based on the filter coefficients to remove the noiseportion of the microphone signal and obtain a processed microphonesignal. The method may include outputting the processed microphonesignal.

In another aspect, an integrated circuit may include a processor that isconfigured to upsample a vibration signal to obtain an upsampledvibration signal. The upsampled vibration signal may have one or moreaxial components. The processor may be configured to determinecorrelation values for the one or more axial components. A respectiveaxial component may have a corresponding correlation value based on amicrophone signal, the upsampled vibration signal, or both. Theprocessor may be configured to determine filter coefficients based onthe upsampled vibration signal if a correlation value is above athreshold. The processor may be configured to filter the axial componentcorresponding to a determined highest correlation value from amongst thecorrelation values of the upsampled vibration signal based on the filtercoefficients to remove a mechanical noise portion of the microphonesignal and obtain a processed microphone signal. The processor may beconfigured to output the processed microphone signal.

In another aspect, a method may include upsampling a vibration signal toobtain an upsampled vibration signal. The upsampled vibration signal mayhave one or more axial components. The method may include determiningcorrelation values for the one or more axial components. A respectiveaxial component may have a corresponding correlation value based on amicrophone signal, the upsampled vibration signal, or both. The methodmay include determining filter coefficients. The filter coefficients maybe determined based on the upsampled vibration signal when a correlationvalue from the correlation values is above a threshold. The method mayinclude filtering the axial component corresponding to a determinedhighest correlation value from amongst the correlation values of theupsampled vibration signal based on the filter coefficients to remove amechanical noise portion of the microphone signal and obtain a processedmicrophone signal. The method may include outputting the processedmicrophone signal.

In another aspect, an image capture device may include a processor thatis configured to upsample a vibration signal to obtain an upsampledvibration signal. The upsampled vibration signal may have one or moreaxial components. The processor may be configured to determinecorrelation values for the one or more axial components. A respectiveaxial component may have a corresponding correlation value. Thecorresponding correlation value may be based on a microphone signal, theupsampled vibration signal, or both. The processor may be configured tofilter the axial component corresponding to a determined highestcorrelation value from amongst the correlation values of the upsampledvibration signal based on filter coefficients to remove a mechanicalnoise portion of the microphone signal and obtain a processed microphonesignal. The processor may be configured to output the processedmicrophone signal.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is best understood from the following detaileddescription when read in conjunction with the accompanying drawings. Itis emphasized that, according to common practice, the various featuresof the drawings are not to-scale. On the contrary, the dimensions of thevarious features are arbitrarily expanded or reduced for clarity.

FIGS. 1A-D are isometric views of an example of an image capture device.

FIGS. 2A-B are isometric views of another example of an image capturedevice.

FIG. 2C is a cross-sectional view of the image capture device of FIGS.2A-B.

FIGS. 3A-B are block diagrams of examples of image capture systems.

FIG. 4A is a diagram of a top-view of an image capture device inaccordance with embodiments of this disclosure.

FIG. 4B is a diagram of a front-view of the image capture device shownin FIG. 4A in accordance with embodiments of this disclosure.

FIG. 4C is a diagram of a rear-view of the image capture device shown inFIG. 4A in accordance with embodiments of this disclosure.

FIG. 5 is a flow diagram of an example of a method for reducingvibration noise.

FIG. 6. is a flow diagram of another example of a method for reducingvibration noise.

FIG. 7 is a block diagram of an example of an integrated circuit forreducing vibration noise.

FIG. 8 is a diagram of example plots of correlation values of microphoneand IMU signals.

DETAILED DESCRIPTION

In the implementations described herein, the level of unwantedstructural vibration noise may be reduced in the captured microphonesignal using data from a vibration sensor. The vibration sensor may beused in conjunction with a microelectromechanical system (MEMS)microphone, an active noise cancellation system, or both. The vibrationsensor may be configured to detect mechanical vibration withoutdetecting acoustic vibration. The detected mechanical vibration may beused as an error signal in an adaptive filter.

FIGS. 1A-D are isometric views of an example of an image capture device100. The image capture device 100 may include a body 102 having a lens104 structured on a front surface of the body 102, various indicators onthe front of the surface of the body 102 (such as LEDs, displays, andthe like), various input mechanisms (such as buttons, switches, andtouch-screen mechanisms), and electronics (e.g., imaging electronics,power electronics, etc.) internal to the body 102 for capturing imagesvia the lens 104 and/or performing other functions. The image capturedevice 100 may be configured to capture images and video and to storecaptured images and video for subsequent display or playback.

The image capture device 100 may include various indicators, includingLED lights 106 and LCD display 108. The image capture device 100 mayalso include buttons 110 configured to allow a user of the image capturedevice 100 to interact with the image capture device 100, to turn theimage capture device 100 on, to operate latches or hinges associatedwith doors of the image capture device 100, and/or to otherwiseconfigure the operating mode of the image capture device 100. The imagecapture device 100 may also include a microphone 112 configured toreceive and record audio signals in conjunction with recording video.The image capture device 100 may also include a drain microphone 112Aconfigured to receive and record audio signals in conjunction withrecording video.

The image capture device 100 may include an I/O interface 114 (e.g.,hidden as indicated using dotted lines). As best shown in FIG. 1B, theI/O interface 114 can be covered and sealed by a removable door 115 ofthe image capture device 100. The removable door 115 can be secured, forexample, using a latch mechanism 115 a (e.g., hidden as indicated usingdotted lines) that is opened by engaging the associated button 110 asshown.

The removable door 115 can also be secured to the image capture device100 using a hinge mechanism 115 b, allowing the removable door 115 topivot between an open position allowing access to the I/O interface 114and a closed position blocking access to the I/O interface 114. Theremovable door 115 can also have a removed position (not shown) wherethe entire removable door 115 is separated from the image capture device100, that is, where both the latch mechanism 115 a and the hingemechanism 115 b allow the removable door 115 to be removed from theimage capture device 100.

The image capture device 100 may also include a speaker 116 integratedinto the body 102 or housing. The front surface of the image capturedevice 100 may include two drainage ports as part of a drainage channel118. The image capture device 100 may include an interactive display 120that allows for interaction with the image capture device 100 whilesimultaneously displaying information on a surface of the image capturedevice 100. As illustrated, the image capture device 100 may include thelens 104 that is configured to receive light incident upon the lens 104and to direct received light onto an image sensor internal to the lens104.

The image capture device 100 of FIGS. 1A-D includes an exterior thatencompasses and protects internal electronics. In the present example,the exterior includes six surfaces (i.e. a front face, a left face, aright face, a back face, a top face, and a bottom face) that form arectangular cuboid. Furthermore, both the front and rear surfaces of theimage capture device 100 are rectangular. In other embodiments, theexterior may have a different shape. The image capture device 100 may bemade of a rigid material such as plastic, aluminum, steel, orfiberglass. The image capture device 100 may include features other thanthose described here. For example, the image capture device 100 mayinclude additional buttons or different interface features, such asinterchangeable lenses, cold shoes and hot shoes that can add functionalfeatures to the image capture device 100, etc.

The image capture device 100 may include various types of image sensors,such as a charge-coupled device (CCD) sensors, active pixel sensors(APS), complementary metal-oxide-semiconductor (CMOS) sensors, N-typemetal-oxide-semiconductor (NMOS) sensors, and/or any other image sensoror combination of image sensors.

Although not illustrated, in various embodiments, the image capturedevice 100 may include other additional electrical components (e.g., animage processor, camera SoC (system-on-chip), etc.), which may beincluded on one or more circuit boards within the body 102 of the imagecapture device 100.

The image capture device 100 may interface with or communicate with anexternal device, such as an external user interface device, via a wiredor wireless computing communication link (e.g., the I/O interface 114).The user interface device may, for example, be the personal computingdevice 360 described below with respect to FIG. 3B. Any number ofcomputing communication links may be used. The computing communicationlink may be a direct computing communication link or an indirectcomputing communication link, such as a link including another device ora network, such as the internet, may be used.

In some implementations, the computing communication link may be a Wi-Filink, an infrared link, a Bluetooth (BT) link, a cellular link, a ZigBeelink, a near field communications (NFC) link, such as an ISO/IEC 20643protocol link, an Advanced Network Technology interoperability (ANT+)link, and/or any other wireless communications link or combination oflinks.

In some implementations, the computing communication link may be an HDMIlink, a USB link, a digital video interface link, a display portinterface link, such as a Video Electronics Standards Association (VESA)digital display interface link, an Ethernet link, a Thunderbolt link,and/or other wired computing communication link.

The image capture device 100 may transmit images, such as panoramicimages, or portions thereof, to the user interface device (not shown)via the computing communication link, and the user interface device maystore, process, display, or a combination thereof the panoramic images.

The user interface device may be a computing device, such as asmartphone, a tablet computer, a phablet, a smart watch, a portablecomputer, and/or another device or combination of devices configured toreceive user input, communicate information with the image capturedevice 100 via the computing communication link, or receive user inputand communicate information with the image capture device 100 via thecomputing communication link.

The user interface device may display, or otherwise present, content,such as images or video, acquired by the image capture device 100. Forexample, a display of the user interface device may be a viewport intothe three-dimensional space represented by the panoramic images or videocaptured or created by the image capture device 100.

The user interface device may communicate information, such as metadata,to the image capture device 100. For example, the user interface devicemay send orientation information of the user interface device withrespect to a defined coordinate system to the image capture device 100,such that the image capture device 100 may determine an orientation ofthe user interface device relative to the image capture device 100.

Based on the determined orientation, the image capture device 100 mayidentify a portion of the panoramic images or video captured by theimage capture device 100 for the image capture device 100 to send to theuser interface device for presentation as the viewport. In someimplementations, based on the determined orientation, the image capturedevice 100 may determine the location of the user interface deviceand/or the dimensions for viewing of a portion of the panoramic imagesor video.

The user interface device may implement or execute one or moreapplications to manage or control the image capture device 100. Forexample, the user interface device may include an application forcontrolling camera configuration, video acquisition, video display, orany other configurable or controllable aspect of the image capturedevice 100.

The user interface device, such as via an application, may generate andshare, such as via a cloud-based or social media service, one or moreimages, or short video clips, such as in response to user input. In someimplementations, the user interface device, such as via an application,may remotely control the image capture device 100 such as in response touser input.

The user interface device, such as via an application, may displayunprocessed or minimally processed images or video captured by the imagecapture device 100 contemporaneously with capturing the images or videoby the image capture device 100, such as for shot framing, which may bereferred to herein as a live preview, and which may be performed inresponse to user input. In some implementations, the user interfacedevice, such as via an application, may mark one or more key momentscontemporaneously with capturing the images or video by the imagecapture device 100, such as with a tag, such as in response to userinput.

The user interface device, such as via an application, may display, orotherwise present, marks or tags associated with images or video, suchas in response to user input. For example, marks may be presented in acamera roll application for location review and/or playback of videohighlights.

The user interface device, such as via an application, may wirelesslycontrol camera software, hardware, or both. For example, the userinterface device may include a web-based graphical interface accessibleby a user for selecting a live or previously recorded video stream fromthe image capture device 100 for display on the user interface device.

The user interface device may receive information indicating a usersetting, such as an image resolution setting (e.g., 3840 pixels by 2160pixels), a frame rate setting (e.g., 60 frames per second (fps)), alocation setting, and/or a context setting, which may indicate anactivity, such as mountain biking, in response to user input, and maycommunicate the settings, or related information, to the image capturedevice 100.

FIGS. 2A-B illustrate another example of an image capture device 200.The image capture device 200 includes a body 202 and two camera lenses204, 206 disposed on opposing surfaces of the body 202, for example, ina back-to-back or Janus configuration.

The image capture device may include electronics (e.g., imagingelectronics, power electronics, etc.) internal to the body 202 forcapturing images via the lenses 204, 206 and/or performing otherfunctions. The image capture device may include various indicators suchas an LED light 212 and an LCD display 214.

The image capture device 200 may include various input mechanisms suchas buttons, switches, and touchscreen mechanisms. For example, the imagecapture device 200 may include buttons 216 configured to allow a user ofthe image capture device 200 to interact with the image capture device200, to turn the image capture device 200 on, and to otherwise configurethe operating mode of the image capture device 200. In animplementation, the image capture device 200 includes a shutter buttonand a mode button. It should be appreciated, however, that, in alternateembodiments, the image capture device 200 may include additional buttonsto support and/or control additional functionality.

The image capture device 200 may also include one or more microphones218A and 218B configured to receive and record audio signals (e.g.,voice or other audio commands) in conjunction with recording video.

The image capture device 200 may include an I/O interface 220 and aninteractive display 222 that allows for interaction with the imagecapture device 200 while simultaneously displaying information on asurface of the image capture device 200.

The image capture device 200 may be made of a rigid material such asplastic, aluminum, steel, or fiberglass. In some embodiments, the imagecapture device 200 described herein includes features other than thosedescribed. For example, instead of the I/O interface 220 and theinteractive display 222, the image capture device 200 may includeadditional interfaces or different interface features. For example, theimage capture device 200 may include additional buttons or differentinterface features, such as interchangeable lenses, cold shoes and hotshoes that can add functional features to the image capture device 200,etc.

FIG. 2C is a cross-sectional view of the image capture device 200 ofFIGS. 2A-B. The image capture device 200 is configured to capturespherical images, and accordingly, includes a first image capture device224 and a second image capture device 226. The first image capturedevice 224 defines a first field-of-view 228 as shown in FIG. 2C andincludes the lens 204 that receives and directs light onto a first imagesensor 230.

Similarly, the second image capture device 226 defines a secondfield-of-view 232 as shown in FIG. 2C and includes the lens 206 thatreceives and directs light onto a second image sensor 234. To facilitatethe capture of spherical images, the image capture devices 224, 226 (andrelated components) may be arranged in a back-to-back (Janus)configuration such that the lenses 204, 206 face in generally oppositedirections.

The fields-of-view 228, 232 of the lenses 204, 206 are shown above andbelow boundaries 236, 238, respectively. Behind the first lens 204, thefirst image sensor 230 may capture a first hyper-hemispherical imageplane from light entering the first lens 204, and behind the second lens206, the second image sensor 234 may capture a secondhyper-hemispherical image plane from light entering the second lens 206.

One or more areas, such as blind spots 240, 242 may be outside of thefields-of-view 228, 232 of the lenses 204, 206 so as to define a “deadzone.” In the dead zone, light may be obscured from the lenses 204, 206and the corresponding image sensors 230, 234, and content in the blindspots 240, 242 may be omitted from capture. In some implementations, theimage capture devices 224, 226 may be configured to minimize the blindspots 240, 242.

The fields-of-view 228, 232 may overlap. Stitch points 244, 246,proximal to the image capture device 200, at which the fields-of-view228, 232 overlap may be referred to herein as overlap points or stitchpoints. Content captured by the respective lenses 204, 206, distal tothe stitch points 244, 246, may overlap.

Images contemporaneously captured by the respective image sensors 230,234 may be combined to form a combined image. Combining the respectiveimages may include correlating the overlapping regions captured by therespective image sensors 230, 234, aligning the captured fields-of-view228, 232, and stitching the images together to form a cohesive combinedimage.

A slight change in the alignment, such as position and/or tilt, of thelenses 204, 206, the image sensors 230, 234, or both, may change therelative positions of their respective fields-of-view 228, 232 and thelocations of the stitch points 244, 246. A change in alignment mayaffect the size of the blind spots 240, 242, which may include changingthe size of the blind spots 240, 242 unequally.

Incomplete or inaccurate information indicating the alignment of theimage capture devices 224, 226, such as the locations of the stitchpoints 244, 246, may decrease the accuracy, efficiency, or both ofgenerating a combined image. In some implementations, the image capturedevice 200 may maintain information indicating the location andorientation of the lenses 204, 206 and the image sensors 230, 234 suchthat the fields-of-view 228, 232, stitch points 244, 246, or both may beaccurately determined, which may improve the accuracy, efficiency, orboth of generating a combined image.

The lenses 204, 206 may be laterally offset from each other, may beoff-center from a central axis of the image capture device 200, or maybe laterally offset and off-center from the central axis. As compared toimage capture devices with back-to-back lenses, such as lenses alignedalong the same axis, image capture devices including laterally offsetlenses may include substantially reduced thickness relative to thelengths of the lens barrels securing the lenses. For example, theoverall thickness of the image capture device 200 may be close to thelength of a single lens barrel as opposed to twice the length of asingle lens barrel as in a back-to-back configuration. Reducing thelateral distance between the lenses 204, 206 may improve the overlap inthe fields-of-view 228, 232.

Images or frames captured by the image capture devices 224, 226 may becombined, merged, or stitched together to produce a combined image, suchas a spherical or panoramic image, which may be an equirectangularplanar image. In some implementations, generating a combined image mayinclude three-dimensional, or spatiotemporal, noise reduction (3DNR). Insome implementations, pixels along the stitch boundary may be matchedaccurately to minimize boundary discontinuities.

FIGS. 3A-B are block diagrams of examples of image capture systems.Referring first to FIG. 3A, an image capture system 300 is shown. Theimage capture system 300 includes an image capture device 310 (e.g., acamera or a drone), which may, for example, be the image capture device100 shown in FIGS. 1A-D or the image capture device 200 shown in FIGS.2A-B.

The image capture device 310 includes a processing apparatus 312 that isconfigured to receive a first image from the first image sensor 314 andreceive a second image from the second image sensor 316. The processingapparatus 312 may be configured to perform image signal processing(e.g., filtering, tone mapping, stitching, and/or encoding) to generateoutput images based on image data from the image sensor 314, imagesensor 316, or both. The image capture device 310 includes acommunications interface 318 for transferring images to other devices.The image capture device 310 includes a user interface 320 to allow auser to control image capture functions and/or view images. The imagecapture device 310 includes a battery 322 for powering the image capturedevice 310. The components of the image capture device 310 maycommunicate with each other via the bus 324.

The processing apparatus 312 may include one or more processors havingsingle or multiple processing cores. The processing apparatus 312 mayinclude memory, such as a random-access memory device (RAM), flashmemory, or another suitable type of storage device such as anon-transitory computer-readable memory. The memory of the processingapparatus 312 may include executable instructions and data that can beaccessed by one or more processors of the processing apparatus 312. Forexample, the processing apparatus 312 may include one or more dynamicrandom access memory (DRAM) modules, such as double data ratesynchronous dynamic random-access memory (DDR SDRAM). In someimplementations, the processing apparatus 312 may include a digitalsignal processor (DSP). In some implementations, the processingapparatus 312 may include an application specific integrated circuit(ASIC). For example, the processing apparatus 312 may include a customimage signal processor.

The first image sensor 314 and the second image sensor 316 may beconfigured to detect light of a certain spectrum (e.g., the visiblespectrum or the infrared spectrum) and convey information constitutingan image as electrical signals (e.g., analog or digital signals). Forexample, the image sensors 314 and 316 may include CCDs or active pixelsensors in a CMOS. The image sensors 314 and 316 may detect lightincident through a respective lens (e.g., a fisheye lens). In someimplementations, the image sensors 314 and 316 include digital-to-analogconverters. In some implementations, the image sensors 314 and 316 areheld in a fixed orientation with respective fields of view that overlap.

The communications interface 318 may enable communications with apersonal computing device (e.g., a smartphone, a tablet, a laptopcomputer, or a desktop computer). For example, the communicationsinterface 318 may be used to receive commands controlling image captureand processing in the image capture device 310. For example, thecommunications interface 318 may be used to transfer image data to apersonal computing device. For example, the communications interface 318may include a wired interface, such as a high-definition multimediainterface (HDMI), a universal serial bus (USB) interface, or a FireWireinterface. For example, the communications interface 318 may include awireless interface, such as a Bluetooth interface, a ZigBee interface,and/or a Wi-Fi interface.

The user interface 320 may include an LCD display for presenting imagesand/or messages to a user. For example, the user interface 320 mayinclude a button or switch enabling a person to manually turn the imagecapture device 310 on and off. For example, the user interface 320 mayinclude a shutter button for snapping pictures.

The battery 322 may power the image capture device 310 and/or itsperipherals. For example, the battery 322 may be charged wirelessly orthrough a micro-USB interface.

The image capture system 300 may implement some or all of the techniquesdescribed in this disclosure, such as the method 800 described in FIG.8.

Referring next to FIG. 3B, another image capture system 330 is shown.The image capture system 330 includes an image capture device 340 and apersonal computing device 360 that communicate via a communications link350. The image capture device 340 may, for example, be the image capturedevice 100 shown in FIGS. 1A-D or the image capture device 200 shown inFIGS. 2A-C. The personal computing device 360 may, for example, be theuser interface device described with respect to FIGS. 1A-D.

The image capture device 340 includes a first image sensor 342 and asecond image sensor 344 that are configured to capture respectiveimages. The image capture device 340 includes a communications interface346 configured to transfer images via the communication link 350 to thepersonal computing device 360.

The personal computing device 360 includes a processing apparatus 362that is configured to receive, using the communications interface 366, afirst image from the first image sensor 342 and a second image from thesecond image sensor 344. The processing apparatus 362 may be configuredto perform image signal processing (e.g., filtering, tone mapping,stitching, and/or encoding) to generate output images based on imagedata from the image sensors 342, 344.

The first image sensor 342 and the second image sensor 344 areconfigured to detect light of a certain spectrum (e.g., the visiblespectrum or the infrared spectrum) and convey information constitutingan image as electrical signals (e.g., analog or digital signals). Forexample, the image sensors 342 and 344 may include CCDs or active pixelsensors in a CMOS. The image sensors 342 and 344 may detect lightincident through a respective lens (e.g., a fisheye lens). In someimplementations, the image sensors 342 and 344 include digital-to-analogconverters. In some implementations, the image sensors 342 and 344 areheld in a fixed relative orientation with respective fields of view thatoverlap. Image signals from the image sensors 342 and 344 may be passedto other components of the image capture device 340 via a bus 348.

The communications link 350 may be a wired communications link or awireless communications link. The communications interface 346 and thecommunications interface 366 may enable communications over thecommunications link 350. For example, the communications interface 346and the communications interface 366 may include an HDMI port or otherinterface, a USB port or other interface, a FireWire interface, aBluetooth interface, a ZigBee interface, and/or a Wi-Fi interface. Forexample, the communications interface 346 and the communicationsinterface 366 may be used to transfer image data from the image capturedevice 340 to the personal computing device 360 for image signalprocessing (e.g., filtering, tone mapping, stitching, and/or encoding)to generate output images based on image data from the image sensors 342and 344.

The processing apparatus 362 may include one or more processors havingsingle or multiple processing cores. The processing apparatus 362 mayinclude memory, such as RAM, flash memory, or another suitable type ofstorage device such as a non-transitory computer-readable memory. Thememory of the processing apparatus 362 may include executableinstructions and data that can be accessed by one or more processors ofthe processing apparatus 362. For example, the processing apparatus 362may include one or more DRAM modules, such as DDR SDRAM.

In some implementations, the processing apparatus 362 may include a DSP.In some implementations, the processing apparatus 362 may include anintegrated circuit, for example, an ASIC. For example, the processingapparatus 362 may include a custom image signal processor. Theprocessing apparatus 362 may exchange data (e.g., image data) with othercomponents of the personal computing device 360 via a bus 368.

The personal computing device 360 may include a user interface 364. Forexample, the user interface 364 may include a touchscreen display forpresenting images and/or messages to a user and receiving commands froma user. For example, the user interface 364 may include a button orswitch enabling a person to manually turn the personal computing device360 on and off. In some implementations, commands (e.g., start recordingvideo, stop recording video, or snap photograph) received via the userinterface 364 may be passed on to the image capture device 340 via thecommunications link 350.

The image capture device 340 and/or the personal computing device 360may be used to implement some or all of the techniques described in thisdisclosure, such as the method 800 of FIG. 8.

FIG. 4A is a diagram of a top-view of an image capture device 400 inaccordance with embodiments of this disclosure. The image capture device400 comprises a camera body 402 having two camera lenses 404, 406structured on front and back surfaces 403, 405 of the camera body 402.The two lenses 404, 406 are oriented in opposite directions and couplewith two images sensors mounted on circuit boards (not shown). Otherelectrical camera components (e.g., an image processor, camera SoC(system-on-chip), etc.) may also be included on one or more circuitboards within the camera body 402 of the image capture device 400.

The lenses 404, 406 may be laterally offset from each other, may beoff-center from a central axis of the image capture device 400, or maybe laterally offset and off-center from the central axis. As compared toan image capture device with back-to-back lenses, such as lenses alignedalong the same axis, the image capture device 400 including laterallyoffset lenses 404, 406 may include substantially reduced thicknessrelative to the lengths of the lens barrels securing the lenses 404,406. For example, the overall thickness of the image capture device 400may be close to the length of a single lens barrel as opposed to twicethe length of a single lens barrel as in a back-to-back configuration.

The image capture device 400 includes a microphone array that comprisesa front-facing component 408, a rear-facing component 412, and aside-facing component 418. The front-facing component 408, therear-facing component 412, and the side-facing component 418 may each bereferred to as a microphone assembly. The side-facing component 418 maybe on any side of the image capture device 400 that is perpendicular tothe front-facing component 408 and the rear-facing component 412, andmay include a top surface, a bottom surface, a left surface, a rightsurface, or any combination thereof. As shown in FIG. 4A, thefront-facing component 408 is disposed on the front surface 403 of theimage capture device. The front-facing component 408 may include one ormore microphone elements 414. The microphone elements 414 may beconfigured such that they are distanced approximately 6 mm to 18 mmapart. The rear-facing component 412 is disposed on the back surface 405of the image capture device 400. The rear-facing component 412 mayinclude one or more microphone elements 416. One or more of themicrophone elements 416 may be configured as a drain microphone. Theside-facing component 418 is shown on a top surface 420 of the imagecapture device 400 in this example. The side-facing component 418 mayinclude one or more microphone elements 422. The microphone elements 422may be configured such that they are distanced approximately 6 mm to 18mm apart. The 6 mm to 18 mm spacing may determine the frequencyresolution of the output. For example, the larger the spacing, the lowerthe highest resolvable frequency. The spacing may be adjusted dependingon the resolution required.

The front-facing component 408, microphone elements 414, rear-facingcomponent 412, and microphone elements 416 are shown in broken lines asthey may not be visible in this view. The front-facing component 408,rear-facing component 412, and side-facing component 418 of themicrophone array may represent microphone elements on an X, Y, Z axis tocreate X, Y, Z components of a First Order Ambisonics B-Format, as shownin FIG. 5. These microphone elements may be oriented on a sphere oroff-axis, and may be transformed to the First Order Ambisonics B-Format.

FIG. 4B is a diagram of a front-view of the image capture device 400shown in FIG. 4A in accordance with embodiments of this disclosure. Asshown in FIG. 4B, the front surface 403 of the image capture device 400comprises the camera lens 404 and the front-facing component 408.Although the front-facing component 408 may include any number ofmicrophone elements, the example shown in FIG. 4B includes threemicrophone elements 414. Each of the microphone elements 414 may beconfigured such that they are distanced approximately 6 mm to 18 mmapart. The side-facing component 418 and the microphone elements 422 areshown in broken lines as they may not be visible in this view.

FIG. 4C is a diagram of a rear-view of the image capture device 400shown in FIG. 4A in accordance with embodiments of this disclosure. Asshown in FIG. 4C, the back surface 405 of the image capture device 400comprises the camera lens 406 and the rear-facing component 412. In anexample, the back surface 405 of the image capture device 400 mayinclude an interactive display 430 that allows for interaction with theimage capture device 400 while simultaneously displaying information ona surface of the image capture device 400. Although the rear-facingcomponent 412 may include any number of microphone elements, the exampleshown in FIG. 4C includes one microphone element 416. In an example, oneor more of the microphone elements 416 may be configured as a drainmicrophone. The side-facing component 418 and the microphone elements422 are shown in broken lines as they may not be visible in this view.

FIG. 5 is a flow diagram of an example of a method 500 for reducingvibration noise. The method 500 may be implemented by an image capturedevice, for example image capture device 100 shown in FIGS. 1A-1D, imagecapture device 400 shown in FIGS. 4A-4C, or both. As shown in FIG. 5,the method 500 includes obtaining 510 a microphone signal. Themicrophone signal may include an acoustic signal portion, a mechanicalnoise portion, or both. The mechanical noise portion may includeunwanted or undesired noise introduced into the microphone signal causedby structural vibrations that are detected by one or more microphonesvia the image capture device body.

The method 500 includes obtaining 520 a vibration signal. The vibrationsignal may be obtained using any vibration sensor such as apiezoelectric vibration sensor or an inertial measurement unit (IMU).Although any vibration sensor may be used, the examples described hereinrefer to the vibration sensor as an IMU for simplicity. The IMU mayinclude one or more components such as an accelerometer, a gyroscope, amagnetometer, or any combination thereof. Each component of the IMU maydetect structural vibration and generate one or more vibration signals.The one or more vibration signals may include respective signalsassociated with an X-axis, Y-axis, Z-axis, or any combination thereof,for each component of the IMU.

Typical sampling rates for vibration sensors are insufficient for noisedetection in the audible bandwidth. For example, a typical sampling ratefor an accelerometer is 200 Hz. In the embodiments disclosed herein, thesampling rates for the vibration sensors are set to overlap with thehuman audible spectrum of about 20 Hz to about 20 kHz. For example, anaccelerometer sampling rate may be set to about 1.6 kHz and a gyroscopesampling rate may be set to about 6.4 kHz.

As shown in FIG. 5, the method 500 includes upsampling 530 the vibrationsignal. Since the vibration signal is obtained as a lower sampling ratethan the microphone signal, the vibration signal is upsampled 530 tomatch the sampling rate of the microphone signal.

The method 500 includes determining 540 a correlation value. Thecorrelation value may be based on the microphone signal and theupsampled vibration signal. A correlation value may be determinedbetween each microphone and each axis of each component of the IMU. Forexample, a device that includes three microphones, an accelerometer, anda gyroscope, 18 correlation values may be determined. The correlationvalues may range from 0 to 1, where a value of 0 would indicate nocorrelation between a microphone signal and a respective vibrationsignal, and a value of 1 would indicate a high correlation between themicrophone signal and the respective vibration signal. An example of thecorrelation between microphone signals and vibration signals is shown inFIG. 8.

Referring again to FIG. 5, in some examples, the method 500 may includedetermining 550 whether a correlation value is above a threshold. Insome examples, the threshold for the correlation value may be 0.5. Ifthe correlation value is determined to be above the threshold, themethod includes determining 560 one or more filter coefficients. Thefilter coefficients may be based on the upsampled vibration signal.

The method 500 includes filtering 570 the upsampled vibration signalbased on the filter coefficients to remove the mechanical noise portionof the microphone signal to obtain a processed microphone signal. Insome examples, the filter coefficient may be applied to the mostcorrelated axis per microphone. For example, if a microphone signal hasa correlation value of 1 associated with an X-axis accelerometer signal,a correlation value of 0.1 associated with a Y-axis accelerometersignal, and a correlation value of 0.4 associated with a Z-axisaccelerometer signal, the filter coefficient may be applied to themicrophone signal associated with the X-axis accelerometer signal. Insome examples, the vibration signal may be a composite signal includingthe X-axis, Y-axis, and Z-axis components associated with the vibrationsignal.

The method 500 includes outputting 580 the processed microphone signal.Outputting 580 the processed microphone signal may include transmittingthe processed microphone signal. Outputting 580 the processed microphonesignal may include storing the microphone signal, for example in amemory such as processing apparatus 312 of FIG. 3A.

FIG. 6. is a flow diagram of another example of a method 600 forreducing vibration noise in microphone signals. The method 600 may beimplemented by an image capture device, for example image capture device100 shown in FIGS. 1A-1D, image capture device 400 shown in FIGS. 4A-4C,or both. As shown in FIG. 6, the method 600 includes obtaining 610 afirst microphone signal and obtaining 615 a second microphone signal.The microphone signals may each include an acoustic signal portion, amechanical noise portion, or both. The mechanical noise portion mayinclude unwanted or undesired noise introduced into the microphonesignal caused by structural vibrations that are detected by one or moremicrophones via the image capture device body.

The method 600 includes obtaining 620 a vibration signal. The vibrationsignal may be obtained using any vibration sensor such as apiezoelectric vibration sensor or an IMU. Although any vibration sensormay be used, the examples described herein refer to the vibration sensoras an IMU for simplicity. The IMU may include one or more componentssuch as an accelerometer, a gyroscope, a magnetometer, or anycombination thereof. Each component of the IMU may detect structuralvibration and generate one or more vibration signals. The one or morevibration signals may include respective signals associated with anX-axis, Y-axis, Z-axis, or any combination thereof, for each componentof the IMU.

Typical sampling rates for vibration sensors are insufficient for noisedetection in the audible bandwidth. For example, a typical sampling ratefor an accelerometer is 200 Hz. In the embodiments disclosed herein, thesampling rates for the vibration sensors are set to overlap with thehuman audible spectrum of about 20 Hz to about 20 kHz. For example, anaccelerometer sampling rate may be set to about 1.6 kHz and a gyroscopesampling rate may be set to about 6.4 kHz.

As shown in FIG. 6, the method 600 includes upsampling 630 the vibrationsignal. Since the vibration signal is obtained as a lower sampling ratethan the microphone signals, the vibration signal is upsampled 630 tomatch the sampling rate of the microphone signals.

The method 600 includes determining 640 a correlation value. Thecorrelation value may be based on the first microphone signal, thesecond microphone signal, and the upsampled vibration signal. Acorrelation value may be determined between each microphone and eachaxis of each component of the IMU. For example, a device that includesthree microphones, an accelerometer, and a gyroscope, 18 correlationvalues may be determined. The correlation values may range from 0 to 1,where a value of 0 would indicate no correlation between a microphonesignal and a respective vibration signal, and a value of 1 wouldindicate a high correlation between the microphone signal and therespective vibration signal. An example of the correlation betweenmicrophone signals and vibration signals is shown in FIG. 8.

Referring again to FIG. 6, in some examples, the method 600 may includedetermining 650 whether a correlation value is above a threshold. Insome examples, the threshold for the correlation value may be 0.5. Ifthe correlation value is determined to be above the threshold, themethod includes determining 660 one or more filter coefficients. Thefilter coefficients may be based on the upsampled vibration signal.

The method 600 includes filtering 670 the upsampled vibration signalbased on the filter coefficients to remove the mechanical noise portionof the microphone signal to obtain one or more processed microphonesignals. In some examples, the filter coefficient may be applied to themost correlated axis per microphone. For example, if a microphone signalhas a correlation value of 1 associated with an X-axis accelerometersignal, a correlation value of 0.1 associated with a Y-axisaccelerometer signal, and a correlation value of 0.4 associated with aZ-axis accelerometer signal, the filter coefficient may be applied tothe microphone signal associated with the X-axis accelerometer signal.

The method 600 includes outputting 680 the processed microphone signals.Outputting 680 the processed microphone signals may include transmittingthe processed microphone signals. Outputting 680 the processedmicrophone signals may include storing the microphone signals, forexample in a memory such as processing apparatus 312 of FIG. 3A.

FIG. 7 is a block diagram of an example of an integrated circuit 700 forreducing vibration noise. The integrated circuit 700 may be implementedin an image capture device, for example image capture device 100 shownin FIGS. 1A-1D, image capture device 400 shown in FIGS. 4A-4C, or both.As shown in FIG. 7, the integrated circuit 700 includes a microphone710, a vibration sensor 720, an upsampler 730, a filter adapter 740, afilter 750, and a summing unit 755. The summing unit 755 may beconfigured to perform an addition operation, a subtraction operation, orboth. The integrated circuit 700 is shown with one microphone and onevibration sensor for simplicity and clarity, and it is understood thatsome implementations may include multiple microphones, multiplevibration sensors, or both. In some implementations, the microphone 710,the vibration sensor 720, or both may be separate from the integratedcircuit 700.

As shown in FIG. 7, the microphone 710 is configured to receive adesired acoustical input 760 from an acoustic source 765 and undesiredmechanical noise 770. The undesired mechanical noise 770 may be causedby a vibration 775. The undesired mechanical noise 770 may be caused bya structural vibration 775 that may be detected by the microphone 710via the image capture device body. The undesired mechanical noise 770may introduce noise into the microphone signal 780.

The vibration sensor 720 is configured to detect the structuralvibration 775. The vibration sensor 720 is configured to receive avibration input 777 caused by the structural vibration 775. Thevibration sensor may include a piezoelectric vibration sensor or an IMU.The IMU may include one or more components such as an accelerometer, agyroscope, a magnetometer, or any combination thereof. Each component ofthe IMU may detect structural vibration and generate one or morevibration signals. The one or more vibration signals may includerespective signals associated with an X-axis, Y-axis, Z-axis, or anycombination thereof, for each component of the IMU.

The upsampler 730 is configured to receive the vibration signal 785 fromthe vibration sensor 720. The upsampler 730 is configured to upsamplethe vibration signal 785. Since the vibration signal 785 may be obtainedat a lower sampling rate than the microphone signal 780, the upsampler730 is configured to upsample the vibration signal 785 to match thesampling rate of the microphone signal 780 and output an upsampledvibration signal 787.

The filter adapter 740 is configured to receive the microphone signal780 and the upsampled vibration signal 787. The filter adapter isconfigured to apply an adaptive algorithm to the microphone signal 780and the upsampled vibration signal 787 to minimize the differencebetween the two signals. An example adaptive algorithm may include anormalized least mean square algorithm. The normalized least mean squarealgorithm may be configured to mimic a desired filter by determining thefilter coefficients that relate to producing the least mean square of anerror signal. The error signal in this example is the difference betweenthe desired signal and the actual signal. The output of the filteradapter 740 is used to update the filter 750. The filter coefficients offilter 750 may be adjusted based on the least mean square result of theupsampled vibration signal and the microphone signal.

The filter 750 is configured to receive the upsampled vibration signal787. The filter 750 is configured to filter the upsampled vibrationsignal 787 to obtain the filtered vibration signal 790. The summing unit755 is configured to remove the filtered vibration signal 790 from themicrophone signal 780 using a subtraction operation and output the errorsignal 795. The error signal 795 may be input to the filter adapter 740to form a feedback loop to continuously update the filter 750. In thisexample, the error signal 795 is the desired signal, i.e. the microphonesignal without vibration noise.

FIG. 8 is a diagram of example plots of correlation values 800 ofmicrophone and IMU signals. In this example, accelerometer-microphonecross-correlation plots 810 and gyroscope-microphone cross-correlationplots 820 for a device configured with three microphones (MIC1, MIC2,MIC 3) are shown. The accelerometer-microphone cross-correlation plots810 show the correlation between the signals of each of MIC 1, MIC 2,and MIC 3 and the accelerometer signals for each of X-axis, Y-axis, andZ-axis of the accelerometer. The accelerometer-microphonecross-correlation plots 810 also show the sum of the signals of eachaccelerometer axis for each microphone. The gyroscope-microphonecross-correlation plots 820 show the correlation between the signals ofeach of MIC 1, MIC 2, and MIC 3 and the gyroscope signals for each ofX-axis, Y-axis, and Z-axis of the gyroscope. The gyroscope-microphonecross-correlation plots 820 also show the sum of the signals of eachgyroscope axis for each microphone.

As shown in FIG. 8, the correlation values may range from −1 to 1, wherea value of 0 indicates no correlation between a microphone signal and arespective vibration signal, a value of 1 indicates a perfectcorrelation between the microphone signal and the respective vibrationsignal, and a value of −1 indicates an inverse correlation or negativecorrelation between the microphone signal and the respective vibrationsignal. As shown in FIG. 8, graph A is a representation of thecross-correlation between the MIC 2 signal and the X-axis accelerometersignal. In graph A, the microphone signal of MIC 2 is shown to have acorrelation value of 1 associated with an X-axis accelerometer signal.Graph B is a representation of the cross-correlation between the MIC 2signal and the Z-axis gyroscope signal. In graph B, the microphonesignal of MIC 2 has a correlation value of about 0.1 associated with aZ-axis gyroscope signal. As shown in FIG. 8, the cross-correlations foreach microphone signal stream may be summed to obtain a compositemicrophone signal. For example, graph C is a representation of thesummed microphone signal stream for MIC 1 that includes the X-axis,Y-axis, and Z-axis components associated with the accelerometer signal.

While the disclosure has been described in connection with certainembodiments, it is to be understood that the disclosure is not to belimited to the disclosed embodiments but, on the contrary, is intendedto cover various modifications and equivalent arrangements includedwithin the scope of the appended claims, which scope is to be accordedthe broadest interpretation so as to encompass all such modificationsand equivalent structures as is permitted under the law.

What is claimed is:
 1. An integrated circuit, comprising: a processorconfigured to: upsample a vibration signal to obtain an upsampledvibration signal, wherein the upsampled vibration signal has one or moreaxial components; determine correlation values for the one or more axialcomponents, wherein a respective axial component has a correspondingcorrelation value based on a microphone signal and the upsampledvibration signal; based on a correlation value from the correlationvalues being above a threshold, determine filter coefficients based onthe upsampled vibration signal; filter the axial component correspondingto a determined highest correlation value from amongst the correlationvalues of the upsampled vibration signal based on the filtercoefficients to obtain a processed microphone signal; and output theprocessed microphone signal.
 2. The integrated circuit of claim 1,wherein the one or more axial components includes an X-axisaccelerometer component, a Y-axis accelerometer component, and a Z-axisaccelerometer component.
 3. The integrated circuit of claim 1, whereinthe one or more axial components includes an X-axis gyroscope component,a Y-axis gyroscope component, and a Z-axis gyroscope component.
 4. Theintegrated circuit of claim 1, wherein the processor is furtherconfigured to: upsample the vibration signal to match a sampling rate ofthe microphone signal.
 5. The integrated circuit of claim 1, wherein theprocessor is further configured to: filter the axial componentcorresponding to the determined highest correlation value to obtain afiltered vibration signal and to remove a noise portion of themicrophone signal.
 6. The integrated circuit of claim 5, wherein theprocessor is further configured to: remove the filtered vibration signalfrom the microphone signal to obtain the processed microphone signal. 7.A method, comprising: upsampling a vibration signal to obtain anupsampled vibration signal, wherein the upsampled vibration signal hasone or more axial components; determining correlation values for the oneor more axial components, wherein a respective axial component has acorresponding correlation value based on a microphone signal and theupsampled vibration signal; determining filter coefficients based on theupsampled vibration signal when a correlation value from the correlationvalues is above a threshold; filtering the axial component correspondingto a determined highest correlation value from amongst the correlationvalues of the upsampled vibration signal based on the filtercoefficients to remove a noise portion of the microphone signal andobtain a processed microphone signal; and outputting the processedmicrophone signal.
 8. The method of claim 7, wherein the one or moreaxial components includes an X-axis accelerometer component, a Y-axisaccelerometer component, and a Z-axis accelerometer component.
 9. Themethod of claim 7, wherein the one or more axial components includes anX-axis gyroscope component, a Y-axis gyroscope component, and a Z-axisgyroscope component.
 10. The method of claim 7, wherein upsampling thevibration signal includes matching a sampling rate of the microphonesignal to obtain the upsampled vibration signal.
 11. The method of claim7, wherein the axial component corresponding to the determined highestcorrelation value is filtered to obtain a filtered vibration signal. 12.The method of claim 11, wherein the filtered vibration signal is removedfrom the microphone signal to obtain the processed microphone signal.13. An image capture device comprising: a processor configured to:upsample a vibration signal to obtain an upsampled vibration signal,wherein the upsampled vibration signal has one or more axial components;determine correlation values for the one or more axial components,wherein a respective axial component has a corresponding correlationvalue based on a microphone signal and the upsampled vibration signal;filter the axial component corresponding to a determined highestcorrelation value from amongst the correlation values of the upsampledvibration signal based on filter coefficients to obtain a processedmicrophone signal; and output the processed microphone signal.
 14. Theimage capture device of claim 13, wherein the filter coefficients arebased on the upsampled vibration signal, and wherein filtering the axialcomponent removes a noise portion of the microphone signal.
 15. Theimage capture device of claim 13, further comprising a vibration sensor.16. The image capture device of claim 15, wherein the vibration sensoris an inertial measurement unit (IMU) that includes an accelerometer, agyroscope, or both.
 17. The image capture device of claim 16, whereinthe vibration sensor includes the accelerometer, and wherein the one ormore axial components includes an X-axis accelerometer component, aY-axis accelerometer component, and a Z-axis accelerometer component.18. The image capture device of claim 16, wherein the vibration sensorincludes the gyroscope, and wherein the one or more axial componentsincludes an X-axis gyroscope component, a Y-axis gyroscope component,and a Z-axis gyroscope component.
 19. The image capture device of claim13, wherein the processor is further configured to: upsample thevibration signal to match a sampling rate of the microphone signal. 20.The image capture device of claim 13, wherein the processor is furtherconfigured to: filter the axial component corresponding to thedetermined highest correlation value to obtain a filtered vibrationsignal; and remove the filtered vibration signal from the microphonesignal to obtain the processed microphone signal.